How Complex Systems Fail

🌈 Abstract

The article discusses the nature of failure in complex systems, such as transportation, healthcare, and power generation. It examines how failure is evaluated, attributed to proximate causes, and the resulting new understanding of patient safety.

🙋 Q&A

[01] The Nature of Failure in Complex Systems

1. What is the key characteristic of complex systems described in the article?

• Complex systems are inherently and unavoidably hazardous by their own nature. The processes involved in these systems are intrinsically and irreducibly hazardous.

2. How do complex systems defend against failure?

• Complex systems have multiple layers of defense against failure, including technical components, human components, and organizational/institutional/regulatory defenses. These defenses normally divert operations away from accidents.

3. Why is it impossible to eliminate all latent failures in complex systems?

• The complexity of these systems makes it impossible for them to run without multiple flaws being present. Eradication of all latent failures is limited by economic cost and the difficulty of identifying how such failures might contribute to an accident before the fact.

4. How do complex systems continue to function despite the presence of many flaws?

• Complex systems run as "broken systems" - they continue to function because they contain so many redundancies and because people can make the system function, despite the presence of many flaws.

[02] Understanding Failure in Complex Systems

1. What is the key insight about the causes of accidents in complex systems?

• There is no isolated "root cause" of an accident. Accidents have multiple contributors, each of which is necessarily insufficient in itself to create an accident. It is the linking of these causes together that creates the circumstances required for the accident.

2. How does hindsight bias affect the analysis of human performance in accidents?

• Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case.

3. How do practitioners balance production demands and the possibility of failure?

• Practitioners operate the system to produce its desired product, while also working to forestall accidents. This dynamic balancing of production and failure prevention is unavoidable, but often misunderstood by outsiders.

[03] Implications for Patient Safety

1. What is the key insight about safety in complex systems?

• Safety is an emergent property of systems; it is not a feature that can be purchased or manufactured separately. The state of safety is always dynamic, as continuous systemic change ensures that hazard and its management are constantly changing.

2. How do practitioners create safety in complex systems?

• Failure-free operations are the result of practitioners' activities to keep the system within tolerable performance boundaries. These adaptations often involve selecting well-rehearsed routines, or creating novel combinations to respond to changing conditions.

3. What is the importance of operators' ability to discern the "edge of the envelope" in complex systems?

• Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where operators can discern the "edge of the envelope" where system performance begins to deteriorate or become difficult to predict.